Time synchronizing of distinct video and data feeds that are delivered in a single mobile IP data network compatible stream

ABSTRACT

Multiple video and/or content feeds are combined and synchronized in real-time into a single mobile IP data network compatible stream for presenting as a coherent display on a mobile device.

CLAIM OF PRIORITY

This application claims the benefit of U.S. Provisional PatentApplication Ser. No. 61/659,709 filed on Jun. 14, 2012, and U.S.Provisional Patent Application Ser. No. 61/785,491 filed on Mar. 14,2013 each of which are hereby incorporated herein by reference in theirentirety.

This application is a continuation-in-part of U.S. Nonprovisionalapplication Ser. No. 11/956,138 entitled CONSUMPTION PROFILE FOR MOBILEMEDIA, filed Dec. 13, 2007 which claims priority to U.S. ProvisionalApplication Ser. No. 60/889,895 entitled MOBILE MEDIA PLATFORM, filedFeb. 14, 2007 and U.S. Provisional Application Ser. No. 60/869,889entitled MOBILE MEDIA PLATFORM, filed Dec. 13, 2006, each of which arehereby incorporated herein by reference in their entirety.

This application is also a continuation-in-part of U.S. Nonprovisionalapplication Ser. No. 11/956,162 entitled SEAMLESSLY SWITCHING AMONGUNICAST, MULTICAST, AND BROADCAST MOBILE MEDIA CONTENT, filed Dec. 13,2007 which claims priority to U.S. Provisional Application Ser. No.60/889,895 entitled MOBILE MEDIA PLATFORM, filed Feb. 14, 2007 and U.S.Provisional Application Ser. No. 60/869,889 entitled MOBILE MEDIAPLATFORM, filed Dec. 13, 2006 each of which are hereby incorporatedherein by reference in their entirety.

This application is also a continuation-in-part of U.S. Nonprovisionalapplication Ser. No. 11/956,184 entitled AUTOMATED CONTENT TAGPROCESSING FOR MOBILE MEDIA, filed Dec. 13, 2007 which claims priorityto U.S. Provisional Application Ser. No. 60/889,895 entitled MOBILEMEDIA PLATFORM, filed Feb. 14, 2007 and U.S. Provisional ApplicationSer. No. 60/869,889 entitled MOBILE MEDIA PLATFORM, filed Dec. 13, 2006,each of which are hereby incorporated herein by reference in theirentirety.

This application is also a continuation-in-part of U.S. Nonprovisionalapplication Ser. No. 11/956,186 entitled MOBILE MEDIA PAUSE AND RESUME,filed Dec. 13, 2007 which claims priority to U.S. ProvisionalApplication Ser. No. 60/889,895 entitled MOBILE MEDIA PLATFORM, filedFeb. 14, 2007 and U.S. Provisional Application Ser. No. 60/869,889entitled MOBILE MEDIA PLATFORM, filed Dec. 13, 2006 each of which arehereby incorporated herein by reference in their entirety.

This application is also a continuation-in-part of U.S. Nonprovisionalapplication Ser. No. 11/956,171 entitled ENCODING AND TRANSCODING FORMOBILE MEDIA, filed Dec. 13, 2007 which claims priority to U.S.Provisional Application Ser. No. 60/889,895 entitled MOBILE MEDIAPLATFORM, filed Feb. 14, 2007 and U.S. Provisional Application Ser. No.60/869,889 entitled MOBILE MEDIA PLATFORM, filed Dec. 13, 2006 each ofwhich are hereby incorporated herein by reference in their entirety.

This application is also a continuation-in-part of U.S. Nonprovisionalapplication Ser. No. 11/956,194 entitled MEDIATION AND SETTLEMENT FORMOBILE MEDIA, filed Dec. 13, 2007 which claims priority to U.S.Provisional Application Ser. No. 60/889,895 entitled MOBILE MEDIAPLATFORM, filed Feb. 14, 2007 and U.S. Provisional Application Ser. No.60/869,889 entitled MOBILE MEDIA PLATFORM, filed Dec. 13, 2006 each ofwhich are hereby incorporated herein by reference in their entirety.

This application is a continuation-in-part of U.S. application Ser. No.12/903,125 entitled DIGITAL RIGHTS MANAGEMENT FOR A MOBILE MEDIAPLATFORM, filed Oct. 12, 2010 which claims priority to U.S. provisionalapplication Ser. No. 61/250,254, entitled DIGITAL RIGHTS MANAGEMENT FORA MOBILE MEDIA PLATFORM, filed Oct. 9, 2009, each of which are herebyincorporated herein by reference in their entirety.

This application is also a continuation-in-part of United StatesNonprovisional application Ser. No. 13/622,681 entitled MEDIA PROCESSOR,filed Sep. 19, 2012 which claims priority to U.S. provisionalapplication Ser. No. 61/536,470, entitled MEDIA PROCESSOR, filed Sep.19, 2011, each of which are hereby incorporated herein by reference intheir entirety.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application incorporates by reference, in its entirety, UnitedStates Nonprovisional application Ser. No. 12/418,576 entitledPROGRESSIVE DOWNLOAD PLAYBACK, filed Apr. 4, 2009.

BACKGROUND

Field

The present invention is related to synchronizing multiple data feedsfor optimized transmission and presentation on a mobile device display.

Description of the Related Art

Multi-feed streaming, such as picture in picture and for providingleaderboard information for display on a mobile device along with avideo of a live event presents challenges to network bandwidth demandand responsiveness to user selection of the individual feeds. Inaddition, a mobile device user often is unable to select how he wouldlike to receive such multiple data feeds when such feeds may be delayedwith respect to one another. Also, coherent display of multiple relatedfeeds that are not synchronized generally cause poor user experience.

SUMMARY

Techniques and applications for delivering multiple video/audio feeds toa mobile device in one stream to be displayed as a Mosaic on one screenare described herein. A device user is enabled to select the audio ofany streamed feed in the Mosaic view or can select to display just oneof the video feeds in the space of the mosaic. Although mobile deviceexamples are used throughout, the methods and systems described hereinmay be similarly applied to other types of displays including mobile OSenabled TVs (e.g. APPLE TV, GOOGLE TV) and the like. In addition,although the examples herein are for touch screen type devices, displaydevices with other user input capabilities (mouse, touch pad, laserpointer, and the like) may be used herein as well.

The methods and systems described herein may include time synchronizeddistinct video feeds (aka channels) in a single mobile device compatiblestream that may include a device specific stream, a videoplayer-specific stream, a Dynamic Adaptive Streaming over HTTP(DASH)-based stream, and the like. The methods and systems describedherein may alternatively include time synchronization of a portion of aset of distinct video feeds in a single mobile device compatible streamso that real-time or “live” feeds may be synchronized.

The methods and systems described herein may include time synchronizedvideo feeds in a single mobile device compatible audio/video stream.

The methods and systems described herein may include time synchronizingdistinct video feeds for delivery to a mobile device via a singleaudio/video stream.

The methods and systems described herein may include time synchronizingdistinct video feeds that are received at a mobile device as a singleaudio/video stream for presentation to the user. In such an embodimentsynchronization may occur on the mobile device based on time-relateddata provided in a side channel and/or metadata associated with thesingle stream.

The methods and systems described herein may include unified display ofdistinct video feeds transmitted via a single audio/video stream over amobile device data network. The single audio/video stream may compriseany number of distinct video feeds and a single audio feed of one of thedistinct video feeds.

The methods and systems described herein may include selectivelydisplaying on a mobile device one or more time synchronized distinctvideo feeds transmitted in a single audio/video stream. The selectionmay be automatic and may be based on location of the mobile device,content subscription associated with the mobile device, identificationof a network carrier providing service to the mobile device, and thelike. In such an embodiment, it may be envisioned that the mobile devicemay receive all of the distinct feeds as one stream and may select asubset of feeds to be presented to the user as described above.

The methods and systems described herein may include time synchronizingselect video feeds from a plurality of available video feeds andencoding the select video feeds into a single mobile device compatiblestream. Selection may be based on location of a mobile device thatrequests streaming of at least one of the select video feeds

The methods and systems described herein may include receiving distinctvideo/audio feeds including meta data for time synchronizing thedistinct feeds into a single mobile-device compatible stream.

Methods and systems described herein may include presenting a multi-viewdisplay in a portion of a device screen of a plurality of feeds thathave been delivered to a mobile device as a single mobile IP datanetwork compatible stream; and in response to a user action associatedwith the multi-view display, presenting a new view of content from atleast one of the plurality of feeds. The user action may be a selectionvia a user interface of the mobile device of a portion of the multi-viewdisplay that includes content from a single feed. The new view mayinclude a single feed that covers substantially the entire multi-viewportion of the device screen.

The methods and systems described herein for streaming a plurality ofvideo feeds as a single mobile IP data network compatible stream andfacilitating user interaction with the plurality of feeds via a coherentdisplay of at least a portion of the video feeds may utilize thefeatures and functionality described in co-pending applicationsincorporated herein by reference. Features and functions described theco-pending applications incorporated herein by reference, such asingestion, encoding/transcoding, mediation, pause and resume, seamlessswitching among unicast/multicast/broadcast, preparation and use of aconsumption profile, automated content tagging, and the like may be usedto identify, ingest, aggregate, stream, and facilitate user interactionwith a plurality of feeds that may include video, audio, content, andthe like.

The methods and systems described herein may include accessing with aserver a plurality of distinct video feeds. The method further includesaccessing time stamp information for at least a portion of the pluralityof distinct video feeds. The method further includes comparing each timestamp information to a reference clock to determine a relative order of,and a relative time difference among, the portion of the plurality ofdistinct video feeds. The method further includes determining asynchronizing point in each of the plurality of distinct video feeds atwhich each feed may be synchronized relative to the reference clock. Themethod further includes combining the plurality of distinct video feedsto form a single video stream comprising a plurality of synchronizeddistinct video feeds based on the synchronizing point. The method mayfurther include encoding the single video stream to form a compositemobile device compatible stream. The method may further includecombining comprises at least one of interleaving, multiplexing, andconcatenating. The combining may include delaying at least one of theplurality of distinct video feeds so that the delayed at least one ofthe plurality of distinct video feeds may be time synchronized with atleast one other of the plurality of distinct video feeds. The step ofcombining the plurality of distinct video feeds may include sub-samplingeach of the distinct video feeds so that the composite mobile devicecompatible stream takes no more bandwidth than one of the distinct videofeeds. The stream may be one of a device specific stream, a videoplayer-specific stream, and a DASH-based stream. The at least one of thedistinct video feeds may include a live event feed. The time stampinformation for the live event feed may be the reference clock, andwherein the plurality of distinct video feeds may be synchronized to thereference clock. The single video stream may include a plurality ofsynchronized distinct video feeds may be an internet protocol streamthat may include metadata for each of the distinct video feeds. Themetadata may facilitate converting the single video stream into thedistinct video feeds for presentation on a mobile device screen. Themetadata may facilitate identifying each of the distinct video feeds toa user interface application of a mobile device on which the distinctvideo feeds are being presented. The single video stream may include aplurality of synchronized distinct video feeds that may include a mosaicof the plurality of the distinct video feeds. The single video streammay include a mosaic that includes position information within themosaic of each distinct video feeds. The single video stream may includea plurality of synchronized distinct video feeds that may include asingle audio feed associated with one of the plurality of synchronizeddistinct video feeds.

The methods and systems described herein may include accessing with aserver a plurality of distinct feeds. The method may include accessing afeed clock associated with a source of at least one of the plurality ofdistinct feeds. The method may include accessing a reference clock. Themethod may include comparing the feed clock to the reference clock todetermine latency between the at least one of the portion of theplurality of distinct feeds and the reference clock. The method mayinclude determining a synchronizing point in each of the plurality ofdistinct feeds based on the determined latency at which each may besynchronized relative to the reference clock. The method may includealigning the synchronizing poit in each of the plurality of distinctfeeds to facilitate delivery of synchronized feeds. The method mayinclude combining the plurality of distinct feeds to form a singlestream including a plurality of synchronized distinct feeds, based onthe determined synchronizing point. In an example, a synchronizationpoint may be a video frame. The method further includes encoding thesingle stream to form a composite mobile device compatible stream. Thestep of combining comprises at least one of interleaving, multiplexing,and concatenating. The step of combining comprises delaying at least oneof the plurality of distinct feeds so that the delayed at least one ofthe plurality of distinct feeds may be time synchronized with at leastone other of the plurality of distinct feeds. The step of combining theplurality of distinct feeds includes sub-sampling each of the distinctfeeds so that the composite mobile device compatible stream takes nomore bandwidth than one of the distinct feeds. The stream may be one ofa device specific stream, a player-specific stream, and a DASH-basedstream. The at least one of the distinct feeds comprises a live eventfeed. The time stamp information for the live event feed may be thereference clock, and wherein the plurality of distinct feeds may besynchronized to the reference clock. The single stream comprising aplurality of synchronized distinct feeds may be an internet protocolstream that comprises metadata for each of the distinct feeds. Thesingle stream comprising a plurality of synchronized distinct feedscomprises a mosaic of the plurality of the distinct video feeds. Thesingle stream comprising a mosaic further comprises position informationwithin the mosaic of each distinct feed. The single stream comprising aplurality of distinct feeds further comprises a plurality of video feedsand a single audio feed associated with one of the plurality of videofeeds.

The methods and systems described herein may include a method includingreceiving a single stream comprising a plurality of distinct feeds. Themethod may include receiving distinct feed data time information in aside channel associated with the stream. The method may includecomparing the time information for each of the distinct feeds todetermine a relative order of, and a relative time difference among, theplurality of distinct feeds. The method further includes determining asynchronizing point in each of the plurality of distinct feeds at whicheach may be synchronized. The method further includes presenting theplurality of distinct feeds synchronized in time based on thesynchronizing point. The synchronizing point in each of the plurality ofdistinct feeds may be associated with a single time value. The step ofpresenting includes delaying at least one of the plurality of distinctfeeds so that the delayed at least one of the plurality of distinctfeeds may be time synchronized with at least one other of the pluralityof distinct feeds. The single stream may be presented as a mosaic of theplurality of the distinct video feeds. The single stream furtherincludes mosaic position information for each distinct video feed. Thesingle stream includes a plurality of distinct feeds further comprises aplurality of video feeds and a single audio feed associated with one ofthe plurality of video feeds.

The methods and systems described herein may include presenting amulti-view display, in a portion of a device screen, of a plurality offeeds that may have been delivered to a mobile device as a single mobileinternet protocol data network compatible stream. The method includespresenting in response to a user action associated with the multi-viewdisplay, a new view of content from at least one of the plurality offeeds. The plurality of feeds may include a plurality of timesynchronized distinct video feeds. The plurality of feeds may include aplurality of distinct video feeds and time information for synchronizingthe plurality of feeds. The time information may be carried in a sidechannel of the single mobile internet protocol data network compatiblestream. The user action may include a selection via a user interface ofthe mobile device of a portion of the multi-view display that mayinclude content from the at least one of the plurality of feeds. The newview may include a single feed of the plurality of feeds that may coversubstantially the entire multi-view display on the device screen.

The methods and systems described herein may include receiving aplurality of feeds from a live event, wherein each feed may beassociated with a feed-specific time stamp. The method includesproviding in a user interface of a mobile device that may be receivingthe feeds at least one interface feature that facilitates selectingbetween holding presentation of a portion of the plurality of feedsuntil an equivalent time stamp from all of the plurality of feeds may bereceived and presenting at least one of the plurality of feeds when itmay be received. The plurality of feeds from a live event includes atleast one video feed of the live event and a data feed derived fromevents at the live event. The data feed may include event real-timeproperties data. The at least one of the plurality of feeds to bepresented when it may be received may be the data feed derived fromevents at the live event. The method further includes providing in theuser interface at least one interface feature that may facilitateselecting between holding presentation of at least one of the pluralityof feeds until an equivalent time stamp from at least one other of theplurality of feeds may be received and presenting at least one of theplurality of feeds when it may be received.

The methods and systems described herein may include receiving aplurality of distinct video feeds and video feed time information thatmay facilitate determining a relative order of the distinct video feeds.The method further includes combining the plurality of distinct videofeeds in the relative order. The method further includes generating amobile device compatible stream from the ordered, combined plurality ofdistinct video feeds by processing the ordered, combined plurality ofdistinct video feeds with a mobile device compatible encoder. The stepof combining may include at least one of interleaving, multiplexing, andconcatenating. The step of combining may include delaying at least oneof the plurality of distinct video feeds so that the delayed at leastone of the plurality of distinct video feeds may be time synchronizedwith at least one other of the plurality of distinct video feeds.

The methods and systems described herein may include accessing with aserver a plurality of distinct video feeds comprising a sequence ofvideo frames. The method further includes accessing time stampinformation for at least a portion of the plurality of distinct videofeeds. The method further includes comparing each of the time stampinformation to a reference clock to determine a relative order of, and arelative time difference among, the portion of the plurality of distinctvideo feeds. The method further includes determining a synchronizingframe in each of the plurality of distinct video feeds at which each maybe synchronized relative to the reference clock. The method furtherincludes combining the video frames of the plurality of distinct videofeeds to form a single video stream comprising a plurality ofsynchronized distinct video feeds based on the determined synchronizingframe. The method further includes encoding the video frames of thesingle video stream to form a composite mobile device compatible stream.The step of combining may include at least one of interleaving,multiplexing, and concatenating. The step of combining may includedelaying at least one of the plurality of distinct video feeds so thatthe delayed at least one of the plurality of distinct video feeds may betime synchronized with at least one other of the plurality of distinctvideo feeds. The step of combining the video frames of the plurality ofdistinct video feeds includes sub-sampling each of the distinct videofeeds so that the composite mobile device compatible stream takes nomore bandwidth than one of the distinct video feeds. The stream may beone of a device specific stream, a video player-specific stream, and aDASH-based stream. The at least one of the distinct video feedscomprises a live event feed. The time stamp information for the liveevent feed may be the reference clock. The plurality of distinct videofeeds may be synchronized to the reference clock. The single videostream may include a plurality of synchronized distinct video feeds maybe an internet protocol stream that comprises metadata for each of thedistinct video feeds. The metadata facilitates converting the singlevideo stream into the distinct video feeds for presentation on a mobiledevice screen. The metadata facilitates identifying each of the distinctvideo feeds to a user interface application of a mobile device on whichthe distinct video feeds may have been presented. The single videostream may include a plurality of synchronized distinct video feedsincludes a mosaic of the plurality of the distinct video feeds. Thesingle video stream may include a mosaic further including positioninformation within the mosaic of each distinct video feed. The singlevideo stream may include a plurality of synchronized distinct videofeeds further including a single audio feed associated with one of theplurality of synchronized distinct video feeds.

The methods and systems described herein may include receiving aplurality of distinct feeds comprising at least one of audio and videoand feed time information that may facilitate determining a relativeorder of the distinct feeds. The method further includes combining theplurality of distinct feeds in the relative order. The method furtherincludes generating a mobile device compatible stream from the ordered,combined plurality of distinct feeds by processing the ordered, combinedplurality of distinct feeds with a mobile device compatible encoder. Thestep of combining may include at least one of interleaving,multiplexing, and concatenating.

The methods and systems described herein may include accessing with aserver a plurality of distinct feeds comprising at least one of audioand video. The method includes accessing time stamp information for atleast a portion of the plurality of distinct feeds. The method furtherincludes comparing each of the time stamp information to a referenceclock to determine a relative order of, and a relative time differenceamong, the portion of the plurality of distinct feeds. The methodfurther includes determining a synchronizing point in each of theplurality of distinct feeds at which each may be synchronized relativeto the reference clock. The method further includes combining theplurality of distinct feeds to form a single stream comprising aplurality of synchronized distinct feeds based on the determinedsynchronizing point. The method further includes encoding the singlestream to form a composite mobile device compatible stream. The step ofcombining includes at least one of interleaving, multiplexing, andconcatenating. The step of combining includes comprises delaying atleast one of the plurality of distinct feeds so that the delayed atleast one of the plurality of distinct feeds may be time synchronizedwith at least one other of the plurality of distinct feeds. The step ofcombining the plurality of distinct feeds includes sub-sampling at leastone of the distinct feeds so that the composite mobile device compatiblestream takes no more bandwidth than one of the distinct feeds. Thestream may be one of a device specific stream, an audio/videoplayer-specific stream, and a DASH-based stream. The at least one of thedistinct feeds comprises a live event feed. The time stamp informationfor the live event feed may be the reference clock. The plurality ofdistinct feeds may be synchronized to the reference clock. The singlestream may include a plurality of synchronized distinct feeds may be aninternet protocol stream that comprises metadata for each of thedistinct feeds. The metadata facilitates converting the single streaminto the distinct feeds for presentation on a mobile device screen. Themetadata facilitates identifying each of the distinct feeds to a userinterface application of a mobile device on which the distinct feeds arebeing presented. The single stream may include a plurality ofsynchronized distinct feeds including a mosaic of the plurality of thedistinct feeds. The single stream includes a mosaic further comprisesposition information within the mosaic of each distinct feed. The singlestream includes a plurality of distinct feeds further including aplurality of video feeds and a single audio feed associated with one ofthe plurality of video feeds.

The methods and systems described herein may include accessing with aserver a plurality of distinct feeds comprising at least one of audioand video and further comprising a sequence of frames. The methodincludes accessing time stamp information for at least a portion of theplurality of distinct feeds. The method includes comparing each of thetime stamp information to a reference clock to determine a relativeorder of, and a relative time difference among, the portion of theplurality of distinct feeds. The method includes determining asynchronizing frame in each of the plurality of distinct feeds at whicheach may be synchronized relative to the reference clock. The methodincludes combining the frames of the plurality of distinct feeds to forma single stream comprising a plurality of synchronized distinct feedsbased on the determined synchronizing frame. The method further includesencoding the frames of the single stream to form a composite mobiledevice compatible stream. The step of combining comprises at least oneof interleaving, multiplexing, and concatenating. The combining includesdelaying at least one of the plurality of distinct feeds so that thedelayed at least one of the plurality of distinct feeds may be timesynchronized with at least one other of the plurality of distinct feeds.The step of combining the frames of the plurality of distinct feedsincludes sub-sampling each of the distinct feeds so that the compositemobile device compatible stream may take no more bandwidth than one ofthe distinct feeds. The stream may be one of a device specific stream, aplayer-specific stream, and a DASH-based stream. The at least one of thedistinct feeds includes a live event feed. The time stamp informationfor the live event feed may be the reference clock. The plurality ofdistinct feeds may be synchronized to the reference clock. The singlestream may includes a plurality of synchronized distinct feeds may be aninternet protocol stream that comprises metadata for each of thedistinct feeds. The single stream includes a plurality of synchronizeddistinct feeds including a mosaic of the plurality of the distinctfeeds. The single stream includes a mosaic further including positioninformation within the mosaic of each distinct feed. The single streammay include a plurality of distinct feeds further including a pluralityof video feeds and a single audio feed associated with one of theplurality of video feeds.

The methods and systems described herein may include receiving in amobile device a single video stream comprising a plurality of timesynchronized distinct video feeds. The method includes accessing, in amemory of the mobile device, a distinct video feed selection criteriathat may indicate one or more of the plurality of time synchronizeddistinct video feeds for presenting to the user. The method includesselecting a portion of the plurality of time synchronized distinct videofeeds received in the single video stream based on the accessed distinctvideo feed selection criteria. The method includes presenting theselected portion to a user of the mobile. The distinct video feedcriteria may be one of location of the mobile device, contentsubscription associated with the mobile device, identification of anetwork carrier providing service to the mobile phone, and userselection of one of the distinct video feeds. The method furtherincludes presenting the selected portion comprises arranging theselected portion into a unified mosaic display. The unified mosaicdisplay includes alternate content for any distinct video feed notincluded in the selected portion.

The methods and systems described herein may include receiving aplurality of distinct video feeds and video feed time information thatfacilitates determining a relative order of the distinct video feeds.The method includes accessing a distinct video feed selection criteriathat indicates one or more of the plurality of time synchronizeddistinct video feeds. The method includes selecting a portion of theplurality of distinct video feeds based on the accessed distinct videofeed selection criteria. The method includes combining the selectedportion of the plurality of distinct video feeds in the relative order.The method includes generating a mobile device compatible stream fromthe ordered, combined plurality of distinct video feeds by processingthe ordered, combined plurality of distinct video feeds with an mobiledevice compatible encoder. The distinct video feed criteria may be oneof location for viewing a distinct video feed, content subscription,identification of a network carrier providing service over which themobile device compatible stream may have to be transmitted, and userselection of one of the distinct video feeds. The step of combining mayinclude at least one of interleaving, multiplexing, and concatenating.The step of may include delaying at least one of the plurality ofdistinct video feeds so that the delayed at least one of the pluralityof distinct video feeds may be time synchronized with at least one otherof the plurality of distinct video feeds. The step of combining theplurality of distinct video feeds includes sub-sampling each of thedistinct video feeds so that the composite mobile device compatiblestream takes no more bandwidth than one of the distinct video feeds. Thestream may be one of a device specific stream, a video player-specificstream, and a DASH-based stream. The at least one of the distinct videofeeds may include a live event feed. The single video stream may includea plurality of synchronized distinct video feeds may be an internetprotocol stream that may include metadata for each of the distinct videofeeds. The single video stream may include a plurality of synchronizeddistinct video feeds including a mosaic of the plurality of the distinctvideo feeds. The single video stream includes a mosaic further includingposition information within the mosaic of each distinct video feed. Thesingle video stream may include a plurality of synchronized distinctvideo feeds further including a single audio feed associated with one ofthe plurality of synchronized distinct video feeds.

The methods and systems described herein may include receiving aselection of a live video feed comprising a plurality of video frames,wherein the live video feed includes time information associated witheach frame. The method includes accessing the time informationassociated with a current frame of the selected live video feed. Themethod further includes querying a server that generates a feed of eventreal-time properties data frames to determine an event real-timeproperties data frame that may correspond to the time informationassociated with the current frame of the selected live video feed. Themethod includes retrieving the determined event real-time propertiesdata frame. The method includes presenting the determined eventreal-time properties data frame with the current frame of selected livevideo feed on an audio/video user interface of a mobile device. The stepof presenting the determined event real-time properties data frameincludes displaying a leaderboard display comprising the event real-timeproperties data in a predefined format. The method further includespresenting subsequent frames of the selected live video feed and theevent real-time properties data until at least one of a new live videofeed may be selected or an error conditions occurs.

The methods and systems described herein may include receiving a userselection through a user interface of a mobile device of a live videofeed. The method further includes receiving the live video feed over amobile device radio network in response to the user selection. Themethod further includes retrieving with the mobile device a latestleaderboard frame that may be periodically generated with a server basedon event real-time properties data received by the server. The methodfurther includes presenting the live video feed as it may be receivedalong with each latest leaderboard frame as it may be retrieved, whereinthe presented latest leaderboard frame lags the presented live videofeed by less than three seconds.

BRIEF DESCRIPTION OF FIGURES

FIG. 1 depicts a user interface for displaying multiple distinct videofeeds on a mobile device that were streamed to the device in a singlestream.

FIG. 2 depicts an embodiment of time synchronizing multiple distinctvideo fees into a single mobile device compatible stream.

FIG. 3 depicts an alternate embodiment of time synchronizing multipledistinct video fees into a single mobile device compatible stream.

FIG. 4 depicts an embodiment of time synchronizing video and eventreal-time properties frames.

FIGS. 5A and 5B depict a live race event application of streaming aplurality of video feeds in a single stream over a mobile devicenetwork.

FIG. 6 depicts a block diagram of select components of a system thatimplements the methods of multi-feed mobile-network single streamingdescribed herein.

FIG. 7 depicts a diagram of an exemplary embodiment of receiving andproviding distinct feeds as individual streams and as a single streamover a mobile device network.

DETAILED DESCRIPTION

The methods and systems described herein improve mobile IP networkbandwidth utilization. Instead of a mobile device needing to decode aseparate stream for each of a plurality of video feeds at once tomaintain a coherent display (and the network having to provide bandwidthfor separate streams), a server ingests all relevant feeds and createsmobile-device compatible stream (e.g. a time synchronized stream) of aselect subset of the ingested feeds for delivery over the mobile IPnetwork. A device receives the single stream and a single player decodesand presents the stream as a single video and audio presentation. Thisreduces network bandwidth demand by a factor roughly equivalent to thenumber of feeds that are combined into one stream and reduces deviceprocessing load by a similar amount (decoding one stream instead ofdecoding and arranging all of the separate streams in one display).

In an example of presenting four games taken from four distinct gamefeeds on one screen of a mobile device, if each game were streamedseparately over a mobile network, each would need 1 Mbps of network anddecoding bandwidth, which equals 4 Mbps bandwidth demand on the networkand device decoding of four streams for display at once. By combining(and optionally time synchronizing) the four game feeds into one stream,the net network and decoding bandwidth required for all four games isapproximately the equivalent of a single separately streamed game or 1Mbps, effectively reducing bandwidth demand by a factor of four.

With such dramatic bandwidth savings, unlocking enhanced functionalitymay result in significant improvements in user interfaces, such as:replacing a conventional simple scrolling text program guide with adynamic scrolling guide that depicts video of the programs; view allnews services on one screen rather than having to switch among newsfeeds; view a range of sports or all games for one sport on a singlescreen; display a user's favorite programs on one screen, and the like.These are merely exemplary improvements in mobile device user interfacecapabilities. It will be apparent that other improvements are alsoenabled through single streaming of a plurality of feeds as describedherein.

Benefits of the methods and systems described herein may extend wellbeyond the reduction in bandwidth demand and processing overhead toaccomplish displaying multiple feeds in a single region of a display.Although a plurality of feeds are streamed to a device as a singlestream, each feed is presented in a portion of a display so that useractions (e.g. through a touchscreen, mouse and the like) within afeed-specific portions of the display screen may result in the userinteracting with the specific feed as if the feed were being streameddirectly to the device. In addition, a user action taken on any sectionof the multi-feed portion of the device display may result in a new viewrelated to the section. Examples described herein include audio control,video control, and the like.

There are many benefits of these techniques. Through the noveltechniques here that result in significant bandwidth conservation,network and processing bandwidth can be allocated for special features(e.g. commercialization of content, and the like). Likewise by requiringminimal device computing resources, even low processing power devicescan incorporate advanced features such as screen-in-screen, videoprogram guide, and the like. Similarly, by requiring less processingpower, power drain is reduced because all of the computations tocoherently combine the feeds are done before the device receives thestream. Beyond the device benefits, network providers gain benefits ofconsistent network load for streaming video to a device that is notdependent on the number of feeds being sourced, thereby alleviatingnetwork congestion for advanced video features like multiple programsbeing displayed simultaneously on a mobile device. When these featuresare combined with selective feed display capabilities in the mobiledevices as described herein, broadcasting a multi-feed stream to bereceived by many devices may further reduce network bandwidth demand.

An exemplary user interface screen is depicted in FIG. 1, which isdescribed here. With a basic touch-screen mobile device, the userinterface for this advanced functionality is surprisingly simple. Oncethe multi-feed/single-stream video functionality is launched, the screencan be filled with multiple feeds. FIG. 1 shows an exemplary five-feedscreen display 102. By touching one of the displayed feeds when an audioselection function is active, the user can select audio for that feed tobe played by the device 104. In this way, the multi-feed single streamuser interface facilitates interactivity and cross combinations of audioassociated with the feeds. By selecting one of the displayed feeds whena feed selection function is active, the user can enable that feed totake over the screen portion of the other feeds to form a single feedvideo 108. At any time the user can return to the multi-feed view toselect another feed or the user can toggle through the feeds in singlefeed mode. These and many other user interface functions and featuresare envisioned and contemplated within the scope of this disclosure.

Examples include streaming live broadcast coverage of an event such asauto racing with multiple views: facing the driver, driver's view,aerial view, real-time statistics, and the like. Another exampleincludes fantasy football in which a fantasy football player's team maybe presented in some of the feeds and video of the games in which theteam members are playing may be presented in other of the feeds. Theuser may be able to select among audio of the game feeds.

Multi-feed streaming may be associated with digital rights management ofthe streamed content. Each feed may be separately managed for digitalrights so that only users with specific rights (e.g. a service plan thatprovides for a real-time statistics view) may view a rights-protectedfeed. Such protection can be accomplished in the mobile device player sothat any unauthorized feed can be discarded or replaced with othercontent during playback. Alternatively DRM protection can beaccomplished in a unicast environment by sending fewer feeds or bysending alternate content (e.g. an ad for the unauthorized feed/channelwith plan pricing) when the target user is not authorized for the feed.Other methods of providing DRM protection and authentication per feedare possible including, geo-location based authentication, network-basedauthentication, and the like. In a live-sports example, certain feeds(e.g. replays) may be available to a mobile user who is located within apredefined radius of the playing field. Alternatively, mobile userswithin the predefined radius may be prohibited from viewing certainreplays similar to what replays are shown on the venue projectionscreens.

Network providers may participate in authentication and permissions toreceive a particular feed in a multi-feed stream. In an example, acarrier may provide high definition content but may limit access to thiscontent to 4G users. Therefore, if a user is not a 4G user, his devicemay not be authorized to receive this content as one of the multi-feeds.The feed may be disabled by the device playback software or it may bereplaced when the stream is prepared with content that the non-4G useris authorized to receive.

Synchronizing the distinct feeds and/or content from other sources thatmay be combined in the single stream or as metadata delivered with thestream may enable coherent display of the images and related information(e.g. real-time statistics, etc.). Synchronizing feeds may be donethrough a variety of techniques including capturing time informationfrom the individual feeds and adjusting the relative timing of the feedsrelative to the stream being delivered to the mobile device as depictedin FIG. 2. Alternatively, time references (e.g. RF satellite timing) maybe used along with feed metadata to determine an appropriate time shiftfor each feed to properly synchronize the feeds into a single datastream to a mobile device as depicted in FIG. 3.

Synchronization in real-time may require satellite based referencesbecause reference timing from the video production trucks generally arenot based on highly accurate clocks. One consideration is that latencyis introduced in the feeds when the feeds are transmitted from thetruck/base station to a satellite and then back to a receiving stationfor delivery to a mobile media platform that provides the multi-feedsingle-streaming described herein. Therefore other time sources may needto be employed for accurate locking of synchronized streams, such as GPStime, Network Time Protocol time, Internet time, venue local time, timereplicated from other content from the same venue or location, timetaken from another video stream, and the like.

One technique of using a reference clock may be to determine the clockthat the feed itself is set to; make sure that the network time is insynch with the video time-video; and then insert the timesynchronization data in a way that may be similar to inserting closecaptioning information.

When video feeds from different cameras at a single event (e.g. a carrace) are to be added to the Mosaic display, the video sources arepreferably synchronized in time before encoding of the mosaic isperformed. This may be important to ensure that common events occurringin both video feeds appear to occur at the same time to the viewer.Therefore, time stamping video as it is captured at the event or atleast when it is transmitted from the event to a video server may berequired. This source-based time stamping also eliminates any need tocompensate for latency of one camera feed relative to the other becauseboth feeds can be locked so that they are fed into encodingsynchronized. The principles of source-based time synching of videofeeds are now described in greater detail.

During live encoding, the encoder may stamp every video frame with adate/time stamp. Local venue time (e.g. GPS time) may be used by theencoder to configure the date/time stamp automatically into the video.This date/time stamp may be received by an appropriate applicationrunning on a mobile device or the like in the header portion of thestream. The header is not viewable by the user, but is discoverable bythe application.

Within the application a listener process may be continuously scanningthe header of the video stream being received to find the date/timestamp. When a user selects one of the videos that are being presented tothe user, the corresponding time stamp will be captured. When thedate/time stamp is captured, the application queries a separate eventreal-time properties (ERP) feed as described herein to find acorresponding date/time frame that is close in time to the captured timestamp. A difference between the date/time frame and the time stamp of afew seconds is typical since it takes about one second to create aleader board frame utilizing event real-time properties data.

Until a corresponding date/time frame is retrieved, the selected videoplays normally. Once a date/time frame that corresponds to the videotime stamp is received, a leaderboard with the pertinent ERP update isdisplayed to the user as a fully synced viewing experience of the ERPdata showing in the leaderboard and the video stream that is playing.Once the two feeds are synchronized, there may be no additional latencybetween the video and the ERP data to deal with. Once the user selects anew video, this process repeats. Note that generally, it will takeapproximately one second for the synced event real-time properties datato load into the application.

Once a time match between ERP data and the video feed is established theapplication continues to monitor that the data is in sync by validatingthe date/time match at less frequent intervals (e.g. ten secondintervals) throughout the video play.

This date/time ERP and video frame synching may be active whenever video(e.g. live video from an event such as a sports event) is beingpresented in the video feed. However, when there is no live video in thefeed or the user has selected a feed that does not include video or hasselected no feed, the application may default to use the most current(low latency/latest frame) event real-time properties data. This isdescribed further with respect to FIG. 4. A similar action may be takenby the application when the user pauses the video, exits the video,starts a new video or if the video fails for some reason. Therefore,when a selected video is playing, the video time stamp is the masterclock to which the other date/time frame data is synchronized. When novideo is playing, the application defaults to using the most currentevent real-time properties data available (e.g. sub-5 second low latencyfeed). If the user stops/pauses the video, the application will switchto display the most current event real-time properties data to the user.

Once the user resumes playback, the application performs the time syncoperation described above so that the user is shown delayed eventreal-time properties data that is time synchronized with the videoplaying. Optionally a message may be sent to the user when thiscondition exists (e.g. resumed playback).

A composite single multi-video feed may also have a time stamp tofacilitate syncing the event real-time properties data. As is describedelsewhere herein, the videos within single multi-video feed are firstsynced to each other and then encoded and wrapped with a video timingheader based on the date/time stamp of a reference or master clock, suchas one generated by or received from the production truck. Thisdate/time stamp is then passed in the composite video feed header to aclient application as noted above and used to synch the event real-timeproperties data similarly to that described above.

A master clock for time stamping video feeds and for synchronizing videoand ERP data may be derived from a clock source at a local venue.Sources of master clock include GPS referenced time, network timeprotocol time (NTP), and the like. Generally video feeds and ERP datagenerated from a single event are time stamped based on the same masterclock. Because video feeds and ERP data may be sourced from twodifferent processing sub systems at an event (e.g. a production truckand a ERP server), clocks in these two sub systems are themselvessynchronized so that each frame from each source can be matched in time.As noted above, a master clock for ERP frame data synchronization may bethe clock time stamp in the video feed header.

In the event of an error condition, certain actions may result to ensurea smooth user viewing experience. In an example error condition, a videofeed may stop unexpectedly. For this condition, as noted above, the ERPdata source will be changed to the most current (low latency) framedata. In another example error condition, a video feed header may notinclude a time stamp. For this condition, the ERP data source becomesthe most current (low latency) frame data. During this condition, thevideo and the ERP leaderboard data may not be synchronized and there maybe no way to determine the latency between the video feed and the ERPdata feed. In yet another example error condition, an ERP data feed maygo down or not be detectable. For this condition, the most recentlyreceived ERP data frame is continuously displayed until the ERP feed isrestored.

Synchronizing a portion of the distinct feeds and/or content from othersources is also envisioned. Selective feed synchronization may beimplemented. A single stream may include feeds and/or content that doesnot necessarily need to be synchronized. In an example, a video programguide may receive feeds from a plurality of channels and may present aportion of these feeds in a display of a mobile device as a videoprogram guide. Other than combining each feed as it is ingested into asingle stream, synchronization among the feeds may not be required. Analternate video program guide embodiment may include presenting aprogram guide along with a larger display of a currently selectedfeed/program. In such an embodiment, the individual video feeds in theprogram guide may not be synchronized to each other but the currentlyselected feed/program in the larger display may be synchronized to itssmaller version in the video program guide.

In another embodiment of selective feed synchronization, a live event,such as an auto race may include feeds from several vantage points inthe race arena (e.g. views from several cars). A single stream of suchan event may include these live feeds as well as content related toleader board details, information about a driver who's view is beingpresented in one of the other feeds, and the like. In such scenarios,the driver views may be synchronized so as to ensure that these displayare coherent in time. However, the non-real-time content of a leaderboard or of driver details may not be synchronized to ensure fine graincoherency. A user may be presented with three driver views and a fourthview of driver details. When a user selects a driver view, the detailsof the driver may be presented in the fourth view.

When using a mobile device at a live event to also view the video feedof the event, some latency between when the live event occurs and whenthat same live event will be presented on the mobile device may beexpected. This may be referred herein to low latency viewing. However,data associated with the live event may be made available before thevideo feed can be propagated from the event through the wired andwireless networks to the user's mobile device. Therefore it may bedesirable to give the user the option of forcing the presentation ofevent data to be delayed so that it can be presented with the video feedof the live event, or allowing the live event data to be presented assoon as it is available. Both options are possible because time stampsare being inserted in both the video feed and the event data feeds sothese time stamps can be used to synchronize presentation if that is thepreferred user experience. Further details regarding low latency viewingare presented below. Although the following embodiments reference a livetrack event, the methods and systems related to low latency viewing canbe applied to any live event.

A first embodiment of low latency viewing may include a sequence ofevents as follows. Note for this embodiment the client app mayproactively ask the server to identify the latest frame of leaderboardinformation.

During a live event, event real-time property messages may becontinuously posted to a video-processing server.

These messages may be processed by accumulating and modifying a fullmodel of a leaderboard for the event by the server. On a one-secondinterval, the state of this model may be written out to a “frame” andthat “frame” may be time-stamped and stored for later retrieval. Thus a“frame” may represent the state of the leaderboard data at a given pointin time. The frame may be created as a file and made available to allclient applications to “pull” data from it.

When the client application starts up its leaderboard view, it may querythe server to determine the latest frame of leaderboard data identifier.The client application may then use this identifier to retrieve thelatest frame, and display it as its leaderboard. The client applicationmay refresh its data on a one-second rate as well.

Since the client application requests the latest frames available, themobile device user may experience a short lag between live eventactivity, and what is displayed in the mobile device leaderboard.However latency includes delays of as much as five seconds associatedwith network time, processing time and content delivery networkinterfacing. Generally, a live event data feed is delivered from theevent to a server in less than one second. Therefore, latency may rangeup to approximately six seconds, but is preferably in the range oftwo-three seconds.

Referring to FIG. 4, which depicts the low latency embodiment describedabove, latencies associated with each aspect of feed generation,ingestion, processing, and delivery are depicted. A sporting event 402,such as an auto race is captured for broadcast and live timing andscoring data 404 is provided to an event real-time properties (ERP)generator 408 that streams ERP data 410 to a media processor 412. Themedia processor 412 generates a series of frames and designates the mostrecently generated as the “latest” frame 414.

Meanwhile a mobile device 418 requests this latest frame 414 from themedia processor 412, which responds with information that the mobiledevice 418 can use to retrieve the latest from 414 from the contentdelivery network (CDN) 420. The mobile device 418 retrieves this latestframe 414 from the CDN 420 and updates it's display of a leader board422. Note that the displayed latest time (e.g. 12:30:45) may be closeto, but generally later than the creation time (e.g. 12:30:42) of thelatest frame 414. In parallel with this mobile device leader boardupdate process, the media processor 412 is updating its frame data bygenerating a next frame 424. In this example, frames are generated onone-second intervals, although other intervals are contemplated andincluded herein.

Depicted along with the CDN 420 is a timeline showing the relativepositioning of each latest and next frame with respect to the currenttime to facilitate depicting a potential latency of two-three secondsfrom event capture to device updating.

A second embodiment of low latency may be no different from the userexperience point of view; however a different implementation can lowerthe latency even further. From a design perspective this feature pushesdata at a regular interval. A sequence of events for this embodiment isas follows:

During a live event, event real-time properties messages are received bythe media processor 412. This data is processed by accumulating andmodifying a full model of the leaderboard. On a configured interval(e.g. multiple times per second), the state of this model is written outto a frame (e.g. as a latest frame 414) in the CDN 420. This frame istime-stamped and stored for later retrieval (e.g. for timesynchronization purposes, or any possible future race replay features).This frame is also pushed out to all listening client devices 418.

When the client application on the mobile device 418 starts up itsleaderboard 422 view it may continuously listen for leaderboard framesthat are pushed to it. When the client application receives a frame itdirectly displays that frame as its leaderboard.

Since the latest frames 414 are always being pushed to the clientapplication, the application does not have to request latest frameinformation and then retrieve it from the CDN, which further reduces thelag between real life and what is displayed in the leaderboard.Therefore, this embodiment bypasses the CDN 420 to further minimize thelatency.

In either of the above embodiments, telemetry data points may be builtinto the leaderboard data frames along with all of the other leaderboarddata points. For both embodiments, the refresh rate for telemetry fieldsmay be 1 second, or the same as a frame generation interval. For thefirst embodiment, the latency of the telemetry fields could be up toapproximately six seconds (taking into account video provider latency aswell). For the later embodiment, the latency of the telemetry fieldscould be up to approximately two seconds (again, taking into accountvideo provider latency).

There may be embodiments in which none of the feeds are synchronized asdescribed herein. Other embodiment may include synchronizing two or moreof the feeds. Yet other embodiments may include synchronizing all of thefeeds that are combined into a single mobile IP network compatiblestream.

There may be embodiments in which feeds in a first stream aresynchronized with feed(s) in a second stream so that screens on multipledevices may be synchronized. One such embodiment may include a live autorace event being broadcast on a first stream and presented on a largescreen that many viewers can view. Individual mobile devices that are inproximity to this screen may receive a multi-feed stream that includesthe large screen feed. The feed that is streamed to the large screen andto the mobile devices may be synchronized by the methods and systemsdescribed herein. In this way mobile device users that are in proximityto the large screen do not experience any delay in the presentation ofthe common feed.

Processing multiple feeds to generate a single stream thereof mayinclude accessing and/or processing metadata and/or source information(e.g. URL) about the feeds. This information may be useful incontrolling display of the feeds, such as when switching betweenmultiple feeds being presented on the device display (e.g. in a Mosaic)to just a single feed being presented or some other combination of feedsbeing presented (e.g. changing from two to three feeds). Other uses ofmetadata may relate to positioning of the feeds in the mosaic, and thelike. Source data, particularly URL data, may be presented or selectableby a user to enable access to the URL content separately from the mosaicinterface functionality.

Multi-feed single-stream video and audio may be compatible with existingIP protocols such as HLS, and future standards such as DASH, and thelike. DASH and the like may facilitate enhanced live video adaptivestreaming such as providing streams in chunks of time (e.g. 10 seconds).These chunks may be further processed by the methods and systems herein,including enhancements to mobile device stream players to split thesechunks into the individual feeds and some contextual information thatthe player recognizes. In an example, the mobile device player may beconfigured to detect contextual information at the three-second point ineach chunk. In an example, information such as an advertisement, leaderboard information, and other content may be included in the stream atthis three-second time and the mobile device player may extract thisinformation and present it appropriately to the user (e.g. an updatedleader board display, and the like.

Synchronization considerations may include startup related effects whena user opens up a screen and starts selecting feeds. Due to networklatency, there may be noticeable delays between when a user selects afeed and it is presented to the user. These delays may extend furtherwhen the feed(s) need to be synchronized. This may be furthercomplicated by the inclusion of alternately sourced content (e.g. aleader board from a website or other such content) that may beseparately updated. Therefore there may be some period of time as a userswitches context (e.g. changing feed selections) that metadata beingprovided with the stream (e.g. to update a leader board) may beperceived by the user to be “out of sync” with the feeds. One option isto delay streaming of data until all of the required data is ready to bestreamed. Using the longest detected latency of the feeds may do this.Alternatively, the mobile device player may participate in thisnon-real-time synchronization and may receive metadata that mayfacilitate the device player providing the proper coordination of livefeeds and non-live content updates. Alternatively video and othercontent may be delivered as one or more sets of files that the mobiledevice player could process and present in a coherent manner.Consideration for enhanced mobile IP streaming standards such as DASH isalso included herein so that capabilities related to IP streaming,containerization, and the like provided by these standards may beeffectively incorporated herein to provide the benefits to the networkprovider and device user as described and envisioned herein.

Referring to FIGS. 5A and 5B, an application of the multi-feedsingle-stream methods and systems described herein is depicted. Videofeeds may be provided via a satellite and distributed through an L-bandRF distribution facility (IRD 1 through 4). Other feeds may be providedfrom the Internet (STRM DECODER 3 through 5). A fifth IRD feed may be anemergency backup feed (IRD 5). All of these feeds may be presented to aVideo Router. A portion of these feeds may be streamed as a singlestream to a user's device to be presented in a coherent display asdescribed herein. The video router may be configured to provideexemplary video routes including:

ROUTE INPUT>OUTPUT

IRLFEED1 PRI IRD1m>APTH_201

IRLFEED1 SEC STRM3>BPTH_201

IRLFEED2 PRI IRD2 m>APTH_202

IRLFEED2 SEC STRM4>BPTH_202

IRLFEED3 IRD3m>APTH_203

IRLFEED4 IRD4m>APTH_204

MOSAIC OUT DEV5_PGM>APTH_205

MOSAIC IN 1 IRD1m>DVE5-1

MOSAIC IN 2 IRD2m>DVE5-2

MOSAIC IN 3 IRD4m>DVE5-3

MOSAIC IN 4 IRD3 m>DEV5-4

Referring to FIG. 6, high-level features of a system for providingmultiple feeds on a single mobile IP network stream are depicted. Thesystem of FIG. 6 may receive 602 a plurality of feeds from sourcesincluding: satellite, fiber, IP, file, tape, and the like. These feedsmay be ingested and delivered to a feed combining engine 604 that mayaccess configuration options 608 that may determine the number of feedsto combine (e.g. 2, 4, 16, 36) and the sources to select. The feedcombining engine 604 may provide a single stream of the combined feedsthat may be provided to an encoder 610 for delivery to a device over amobile IP network.

Referring to FIG. 7, a plurality of feeds 702 may be received andprocessed by an ingestion engine 704. The ingestion engine 704 may alsocombine the feeds 702 into a single stream 708 that may be presented tothe user as a mosaic 710. The ingestion engine 704 may provide thiscombined stream 708 and individual streams 712 of each of the feeds toencoders 714 for delivery to mobile device(s) over a mobile IP network.

Combining the streams may be accomplished by, for example, creating asingle stream that consists of packets from a plurality of otherstreams, such as by interleaving, multiplexing, or concatenating suchpackets and retaining information for managing or tracking the combinedstreams, by multiplexing or otherwise combining signals from differentstreams, or other techniques for combining data streams. One method ofcombining includes scaling each feed to a smaller representative imagethat can be displayed in a mosaic as depicted in at least FIG. 1 hereinbefore combining the scaled feeds into a single video while retainingindividual identifiers of each feed and its location in the mosaic byway of meta-data that is used by the device to manage a particular feed(e.g. allowing a user to select a feed).

Another exemplary method of combining includes multiplexing multiplefeeds into one video to produce a new video source that can be sent toan encoder to create a single channel (stream) with multiple pieces ofcontent. In addition to combining feeds and generating a single video ofthe combined feeds, each individual feed may be delivered to an encoderso that when a user selects one of the feeds from the single stream, alarger (e.g. full screen) video of the selected feed can be streamed tothe user's device without undue delay.

Although references herein include scaling and/or combining feeds priorto encoding, an encoding engine may be adapted to receive multiplesource feeds, scale each as needed for the desired mosaic display, andcreate a single encoded stream to be delivered to a mobile device.Likewise, combining could be performed post encoding by each feed beingsent to an encoder where the feed would be scaled to accommodate amosaic display and a content delivery engine would combine the scaledencoded outputs into a single stream for delivery to a user device. Avideo player/application on the device could receive the stream andgenerate the mosaic from the distinct feeds in the stream.

The methods and systems described herein may be deployed in part or inwhole through a machine that executes computer software, program codes,and/or instructions on a processor. The processor may be part of aserver, cloud server, client, network infrastructure, mobile computingplatform, stationary computing platform, or other computing platform. Aprocessor may be any kind of computational or processing device capableof executing program instructions, codes, binary instructions and thelike. The processor may be or include a signal processor, digitalprocessor, embedded processor, microprocessor or any variant such as aco-processor (math co-processor, graphic co-processor, communicationco-processor and the like) and the like that may directly or indirectlyfacilitate execution of program code or program instructions storedthereon. In addition, the processor may enable execution of multipleprograms, threads, and codes. The threads may be executed simultaneouslyto enhance the performance of the processor and to facilitatesimultaneous operations of the application. By way of implementation,methods, program codes, program instructions and the like describedherein may be implemented in one or more thread. The thread may spawnother threads that may have assigned priorities associated with them;the processor may execute these threads based on priority or any otherorder based on instructions provided in the program code. The processormay include memory that stores methods, codes, instructions and programsas described herein and elsewhere. The processor may access a storagemedium through an interface that may store methods, codes, andinstructions as described herein and elsewhere. The storage mediumassociated with the processor for storing methods, programs, codes,program instructions or other type of instructions capable of beingexecuted by the computing or processing device may include but may notbe limited to one or more of a CD-ROM, DVD, memory, hard disk, flashdrive, RAM, ROM, cache and the like.

A processor may include one or more cores that may enhance speed andperformance of a multiprocessor. In embodiments, the process may be adual core processor, quad core processors, other chip-levelmultiprocessor and the like that combine two or more independent cores(called a die).

The methods and systems described herein may be deployed in part or inwhole through a machine that executes computer software on a server,client, firewall, gateway, hub, router, or other such computer and/ornetworking hardware. The software program may be associated with aserver that may include a file server, print server, domain server,internet server, intranet server, cloud server, and other variants suchas secondary server, host server, distributed server and the like. Theserver may include one or more of memories, processors, computerreadable media, storage media, ports (physical and virtual),communication devices, and interfaces capable of accessing otherservers, clients, machines, and devices through a wired or a wirelessmedium, and the like. The methods, programs or codes as described hereinand elsewhere may be executed by the server. In addition, other devicesrequired for execution of methods as described in this application maybe considered as a part of the infrastructure associated with theserver.

The server may provide an interface to other devices including, withoutlimitation, clients, other servers, printers, database servers, printservers, file servers, communication servers, distributed servers,social networks, and the like. Additionally, this coupling and/orconnection may facilitate remote execution of program across thenetwork. The networking of some or all of these devices may facilitateparallel processing of a program or method at one or more locationwithout deviating from the scope of the invention. In addition, any ofthe devices attached to the server through an interface may include atleast one storage medium capable of storing methods, programs, codeand/or instructions. A central repository may provide programinstructions to be executed on different devices. In thisimplementation, the remote repository may act as a storage medium forprogram code, instructions, and programs.

The software program may be associated with a client that may include afile client, print client, domain client, internet client, intranetclient and other variants such as secondary client, host client,distributed client and the like. The client may include one or more ofmemories, processors, computer readable media, storage media, ports(physical and virtual), communication devices, and interfaces capable ofaccessing other clients, servers, machines, and devices through a wiredor a wireless medium, and the like. The methods, programs or codes asdescribed herein and elsewhere may be executed by the client. Inaddition, other devices required for execution of methods as describedin this application may be considered as a part of the infrastructureassociated with the client.

The client may provide an interface to other devices including, withoutlimitation, servers, other clients, printers, database servers, printservers, file servers, communication servers, distributed servers andthe like. Additionally, this coupling and/or connection may facilitateremote execution of program across the network. The networking of someor all of these devices may facilitate parallel processing of a programor method at one or more location without deviating from the scope ofthe invention. In addition, any of the devices attached to the clientthrough an interface may include at least one storage medium capable ofstoring methods, programs, applications, code and/or instructions. Acentral repository may provide program instructions to be executed ondifferent devices. In this implementation, the remote repository may actas a storage medium for program code, instructions, and programs.

The methods and systems described herein may be deployed in part or inwhole through network infrastructures. The network infrastructure mayinclude elements such as computing devices, servers, routers, hubs,firewalls, clients, personal computers, communication devices, routingdevices and other active and passive devices, modules and/or componentsas known in the art. The computing and/or non-computing device(s)associated with the network infrastructure may include, apart from othercomponents, a storage medium such as flash memory, buffer, stack, RAM,ROM and the like. The processes, methods, program codes, instructionsdescribed herein and elsewhere may be executed by one or more of thenetwork infrastructural elements.

The methods, program codes, and instructions described herein andelsewhere may be implemented on a cellular network having multiplecells. The cellular network may either be frequency division multipleaccess (FDMA) network or code division multiple access (CDMA) network.The cellular network may include mobile devices, cell sites, basestations, repeaters, antennas, towers, and the like. The cell networkmay be a GSM, GPRS, 3G, EVDO, mesh, or other networks types.

The methods, programs codes, and instructions described herein andelsewhere may be implemented on or through mobile devices. The mobiledevices may include navigation devices, cell phones, mobile phones,mobile personal digital assistants, laptops, palmtops, netbooks, pagers,electronic books readers, music players and the like. These devices mayinclude, apart from other components, a storage medium such as a flashmemory, buffer, RAM, ROM and one or more computing devices. Thecomputing devices associated with mobile devices may be enabled toexecute program codes, methods, and instructions stored thereon.Alternatively, the mobile devices may be configured to executeinstructions in collaboration with other devices. The mobile devices maycommunicate with base stations interfaced with servers and configured toexecute program codes. The mobile devices may communicate on a peer topeer network, mesh network, or other communications network. The programcode may be stored on the storage medium associated with the server andexecuted by a computing device embedded within the server. The basestation may include a computing device and a storage medium. The storagedevice may store program codes and instructions executed by thecomputing devices associated with the base station.

The computer software, program codes, and/or instructions may be storedand/or accessed on machine readable media that may include: computercomponents, devices, and recording media that retain digital data usedfor computing for some interval of time; semiconductor storage known asrandom access memory (RAM); mass storage typically for more permanentstorage, such as optical discs, forms of magnetic storage like harddisks, tapes, drums, cards and other types; processor registers, cachememory, volatile memory, non-volatile memory; optical storage such asCD, DVD; removable media such as flash memory (e.g. USB sticks or keys),floppy disks, magnetic tape, paper tape, punch cards, standalone RAMdisks, Zip drives, removable mass storage, off-line, and the like; othercomputer memory such as dynamic memory, static memory, read/writestorage, mutable storage, read only, random access, sequential access,location addressable, file addressable, content addressable, networkattached storage, storage area network, bar codes, magnetic ink, and thelike.

The methods and systems described herein may transform physical and/oror intangible items from one state to another. The methods and systemsdescribed herein may also transform data representing physical and/orintangible items from one state to another.

The elements described and depicted herein, including in flow charts andblock diagrams throughout the figures, imply logical boundaries betweenthe elements. However, according to software or hardware engineeringpractices, the depicted elements and the functions thereof may beimplemented on machines through computer executable media having aprocessor capable of executing program instructions stored thereon as amonolithic software structure, as standalone software modules, or asmodules that employ external routines, code, services, and so forth, orany combination of these, and all such implementations may be within thescope of the present disclosure. Examples of such machines may include,but may not be limited to, personal digital assistants, laptops,personal computers, mobile phones, other handheld computing devices,medical equipment, wired or wireless communication devices, transducers,chips, calculators, satellites, tablet PCs, electronic books, gadgets,electronic devices, devices having artificial intelligence, computingdevices, networking equipments, servers, routers and the like.Furthermore, the elements depicted in the flow chart and block diagramsor any other logical component may be implemented on a machine capableof executing program instructions. Thus, while the foregoing drawingsand descriptions set forth functional aspects of the disclosed systems,no particular arrangement of software for implementing these functionalaspects should be inferred from these descriptions unless explicitlystated or otherwise clear from the context. Similarly, it will beappreciated that the various steps identified and described above may bevaried, and that the order of steps may be adapted to particularapplications of the techniques disclosed herein. All such variations andmodifications are intended to fall within the scope of this disclosure.As such, the depiction and/or description of an order for various stepsshould not be understood to require a particular order of execution forthose steps, unless required by a particular application, or explicitlystated or otherwise clear from the context.

The methods and/or processes described above, and steps thereof, may berealized in hardware, software or any combination of hardware andsoftware suitable for a particular application. The hardware may includea general purpose computer and/or dedicated computing device or specificcomputing device or particular aspect or component of a specificcomputing device. The processes may be realized in one or moremicroprocessors, microcontrollers, embedded microcontrollers,programmable digital signal processors or other programmable device,along with internal and/or external memory. The processes may also, orinstead, be embodied in an application specific integrated circuit, aprogrammable gate array, programmable array logic, or any other deviceor combination of devices that may be configured to process electronicsignals. It will further be appreciated that one or more of theprocesses may be realized as a computer executable code capable of beingexecuted on a machine readable medium.

The computer executable code may be created using a structuredprogramming language such as C, an object oriented programming languagesuch as C++, or any other high-level or low-level programming language(including assembly languages, hardware description languages, anddatabase programming languages and technologies) that may be stored,compiled or interpreted to run on one of the above devices, as well asheterogeneous combinations of processors, processor architectures, orcombinations of different hardware and software, or any other machinecapable of executing program instructions.

Thus, in one aspect, each method described above and combinationsthereof may be embodied in computer executable code that, when executingon one or more computing devices, performs the steps thereof. In anotheraspect, the methods may be embodied in systems that perform the stepsthereof, and may be distributed across devices in a number of ways, orall of the functionality may be integrated into a dedicated, standalonedevice or other hardware. In another aspect, the means for performingthe steps associated with the processes described above may include anyof the hardware and/or software described above. All such permutationsand combinations are intended to fall within the scope of the presentdisclosure.

While the invention has been disclosed in connection with the preferredembodiments shown and described in detail, various modifications andimprovements thereon will become readily apparent to those skilled inthe art. Accordingly, the spirit and scope of the present invention isnot to be limited by the foregoing examples, but is to be understood inthe broadest sense allowable by law.

All documents referenced herein are hereby incorporated herein in theirentirety by reference.

The invention claimed is:
 1. A method comprising: accessing with aserver a plurality of distinct video feeds; accessing time stampinformation for at least a portion of the plurality of distinct videofeeds; determining by comparing each time stamp information to areference clock a relative order of, and a relative time differenceamong, the portion of the plurality of distinct video feeds; determininga synchronizing point in each of the plurality of distinct video feedsat which each is synchronized relative to the reference clock; andcombining the plurality of distinct video feeds to form a single streamcomprising a plurality of synchronized distinct feeds based on thedetermined synchronizing point, the single stream including a pluralityof video feeds and a single audio feed associated with one of theplurality of video feeds, and wherein each of the plurality of distinctvideo feeds are separately managed such that only users with specificrights may view a rights-protected feed.
 2. The method of claim 1,further comprising encoding the single video stream to form a compositemobile device compatible stream, wherein combining the plurality ofdistinct video feeds includes sub-sampling each of the distinct videofeeds so that the composite mobile device compatible stream takes nomore bandwidth than one of the distinct video feeds.
 3. The method ofclaim 1, wherein the single video stream comprising a plurality ofsynchronized distinct video feeds comprises a mosaic of the plurality ofthe distinct video feeds.
 4. The method of claim 1, further comprisingreplacing the rights-protected feed for an unauthorized user in thesingle stream with other content.
 5. The method of claim 1, furthercomprising replacing the rights-protected feed for an unauthorized userin the single stream with an advertisement.
 6. The method of claim 1,wherein at least one of the plurality of distinct video feeds providescontent associated with another one of the one of the plurality ofdistinct video feeds, the at least one of the plurality of distinctvideo feeds is not synchronized to ensure fine grain coherency.
 7. Amethod comprising: accessing with a server a plurality of distinctfeeds; accessing a feed clock associated with a source of at least oneof the plurality of distinct feeds; accessing a reference clock;determining by comparing the feed clock to the reference clock latencybetween the at least one of the portion of the plurality of distinctfeeds and the reference clock; determining a synchronizing point in eachof the plurality of distinct feeds based on the determined latency atwhich each is synchronized relative to the reference clock; aligning thesynchronizing point in each of the plurality of distinct feeds tofacilitate delivery of synchronized feeds; and combining the pluralityof distinct feeds to form a single stream comprising a plurality ofsynchronized distinct feeds, based on the determined synchronizingpoint, the single stream including a plurality of video feeds and asingle audio feed associated with one of the plurality of video feeds,and wherein each of the plurality of distinct video feeds are separatelymanaged such that only users with specific rights may view arights-protected feed.
 8. The method of claim 7, wherein asynchronization point is a video frame.
 9. The method of claim 7,further comprising encoding the single video stream to form a compositemobile device compatible stream, wherein combining the plurality ofdistinct feeds includes sub-sampling each of the distinct feeds so thatthe composite mobile device compatible stream takes no more bandwidththan one of the distinct feeds.
 10. The method of claim 7, wherein thesingle stream comprising a plurality of synchronized distinct feedscomprises a mosaic of the plurality of the distinct video feeds.
 11. Themethod of claim 7, further comprising replacing the rights-protectedfeed for an unauthorized user in the single stream with other content.12. The method of claim 7, further comprising replacing therights-protected feed for an unauthorized user in the single stream withan advertisement.
 13. The method of claim 7, wherein at least one of theplurality of distinct feeds provides content associated with another oneof the one of the plurality of distinct feeds, the at least one of theplurality of distinct feeds is not synchronized to ensure fine graincoherency.
 14. A method comprising: receiving a single stream comprisinga plurality of distinct feeds, the single stream including a pluralityof video feeds and a single audio feed associated with one of theplurality of video feeds, wherein at least one of the plurality ofdistinct video feeds are separately managed for digital rights, andwherein each of the plurality of distinct video feeds are separatelymanaged such that only users with specific rights may view arights-protected feed; receiving distinct feed data time information ina side channel associated with the stream; determining by comparing thetime information for each of the distinct feeds a relative order of, anda relative time difference among, the plurality of distinct feeds;determining a synchronizing point in each of the plurality of distinctfeeds at which each is synchronized; and presenting the plurality ofdistinct feeds synchronized in time based on the synchronizing point.15. The method of claim 14, wherein the synchronizing point in each ofthe plurality of distinct feeds is associated with a single time value.16. The method of claim 14, wherein the single stream is presented as amosaic of the plurality of the distinct video feeds.
 17. The method ofclaim 14, further comprising presenting other content for therights-protected feed for an unauthorized user in the single videostream.
 18. The method of claim 14, further comprising presenting anadvertisement for the rights-protected feed for an unauthorized user inthe single video stream.
 19. The method of claim 14, wherein at leastone of the plurality of distinct feeds provides content associated withanother one of the one of the plurality of distinct feeds, the at leastone of the plurality of distinct feeds is not synchronized to ensurefine grain coherency.